Improving Cross-Validation Classifier Selection Accuracy through Meta-Learning

In order to choose from the large number of classification methods available for use, cross-validation error estimates are often employed. We present this cross-validation selection strategy in the framework of meta-learning and show that conceptually, metalearning techniques could provide better classifier selections than traditional cross-validation selection. Using various simulation studies we illustrate and discuss this possibility. Through a collection of datasets resembling real-world data, we investigate whether these improvements could possibly exist in the real-world as well. Although the approach presented here currently requires significant investment when applied to practical applications, the concept of being able to outperform cross-validation selection opens the door to new classifier selection strategies.